Depth Separation and Weight-width Trade-offs for Sigmoidal Neural Networks
نویسندگان
چکیده
Recent work has shown strong separation between the expressive power of depth2 and depth-3 neural networks. These separation results exhibit a function and an input distributions, so that the function is well-approximable in L2-norm on the input distribution by a depth-3 neural network of polynomial size but any depth-2 neural network that well-approximates it requires exponential size. A limitations of these results is that they work only for certain careful choices of functions and input distributions that are arguably not natural enough. We provide a simple proof of L2-norm separation between the expressive power of depth-2 and depth-3 sigmoidal neural networks for a large class of input distributions, assuming their weights are polynomially bounded. Our proof is simpler than previous results, uses known low-degree multivariate polynomial approximations to neural networks, and gives the first depth-2-vs-depth-3 separation that works for a large class of input distributions.
منابع مشابه
A one-model approach for Computation of congestion With productions trade-offs and weight restrictions
متن کامل
Ethical Perspective: Five Unacceptable Trade-offs on the Path to Universal Health Coverage
This article discusses what ethicists have called “unacceptable trade-offs” in health policy choices related to universal health coverage (UHC). Since the fiscal space is constrained, trade-offs need to be made. But some trade-offs are unacceptable on the path to universal coverage. Unacceptable choices include, among other examples from low-income countries, to expand coverage for services wit...
متن کاملDepth Separation in ReLU Networks for Approximating Smooth Non-Linear Functions
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-linear functions which are radial with respect to the L1 norm; and smooth non-li...
متن کاملDepth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-linear functions which are radial with respect to the L1 norm; and smooth non-li...
متن کاملApplication of multilayer perceptron neural network and support vector machine for modeling the hydrodynamic behavior of permeable breakwaters with porous core
In this research, the application of multilayer perceptron (MLP) neural networks and support vector machine (SVM) for modeling the hydrodynamic behavior of Permeable Breakwaters with Porous Core has been investigated. For this purpose, experimental data have been used on the physical model to relate the reflection and transition coefficients of incident waves as the output parameters to the wid...
متن کامل